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At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 
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VIRTUAL PRIVATE NETWORK ON APPLICATION GATEWAY 
Backgroimd of the Invention 

Field of the Invention 

The present invention pertains generally to network commimicaiions, and 
in particular to a system and method for securely transferring infonnation 
between firewalls over an unprotected network. 

Raclcgmmid TnfnrmatioTi 

Firewalls have become an inoeasingly important part of network design. 
Firewalls provide protection of valuable resources on a private network while 
allowing communication and access with systems located on an utqjrotected 
network such as the Internet. In addition, they operate to block attacks on a 
private network arriving from the unprotected network by providing a single 
connection with limited services. A well designed firewall limits the security 
problems of an Internet connection to a single firewall computer system. This 
allows an organization to focus their network securi^ efforts on the definition of 
the securiQr policy enf(urced by tiie firewall. An nrample of a firewall is given in 
"SYSTEM AND METHOD FOR PROVIDING SECURE INTERNETWORK 
SERVICES" by Boebert et al. (PCT PubUshed Application No. WO 96/131 13, 
published on May 2, 1996), the description of which is hereby incorporated by 
reference. Another description of a firewall is provided by Dan Thomsen in 
"Type Enforcement: the new security model". Proceedings: Multimedia: Full- 
Service Impact on Business, Education, and the Home, SPIE Vol. 2617. p. 143, 
August 1996. Yet another such system is described in "SYSTEM AND 
METHOD FOR ACHIEVING NETWORK SEPARATION" by Gooderum et al. 
(PCT Published Applicaficm No. WO 97/29413, published on August 1 4, 1 997), 
the description of v4uch is haeby incoipoiated by reference. All the above 
systems are examples of application level gateways. Application level gateways 
use proxies or other such mechanisms operating at the ^plication layer to 
process traffic tiirough the firewall. As such, they can review not only the 



message traffic but also message content In addition, they provide 
authentication and identification services, access control and auditing. 

Data to be liansferred on uiqnotected networks like the Internet is 
susceptible to electronic eavesdropping and accidental (or deliberate) corruption. 
Although a firewall can protect data within a private network ftom attacks 
launched from the unprotected network, even that data is vutaerable to both 
eavesdropping and corruption when transferred from the private network to an 
external machine. To address this danger, the Inieniet Engineering Task Force 
(IETF) developed a standard for protecting data transferred between firewalls 
over an unprotected network. The Internet Protocol Security (IPSEC) standard 
calls for encrypting data before it leaves the firet firewall, and then decrypting 
the data when it is received by the second firewall. The decrypted data is then 
delivered to its destination, usually a user workstation connected to the second 
firewall. For this reason IPSEC encryption is sometimes aHlcd fir ewall-to- 
firewall encryption (FEE) and the connection between a workstation connected 
to the first firewall and a client or server connected to the second firewall is 
termed a virtual private network, or VPN. 

The two main components of IPSEC security are data encryption and 
sender authentication. Data encryption increases the cost and time required for 
the eavesdropping party to read the transmitted data. Sender authentication 
ensures that the destination system can verify whedier or not the encrypted data 
was actually sent from the woricstation that it was sxq>posed to be sent from. The 
IPSEC standard dcfmes an encapsulated payload (ESP) as the mechanism used 
to transfer encrypted data. The standard defines an aathentipation header (AH) 
as the mechanism for establishing the sendmg workstation's identity. 

Through the proper use of encryption, the problems of eavesdropping and 
corruption can be avoided; m effect, a protected connection is established from 
the internal network connected to one firewaU through to an internal network 
connected to the second firewaU. In addition. IPSEC can be used to provide a 
protected connection to an external computing system such as a portable 
personal computer. 
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IPSEC enoyption and deciyptioii woik within die IP layer of the network 
protocol stack. This means that all communication between two IP addresses 
wiU be protected because all interfirewall communication must go through the IP 
layer. Such an approach is preferable over encryption and decryption at higher 
5 levels in the network protocol stack since when encryption is performed at layers 
higher than the IP layer more work is required to ensure lhat all supported 
communication is properly protected. In addition, since IPSEC encryption is 
handled below the Transport layer, IPSEC can encrypt data sent by any 
appUcation. IPSEC therefore becomes a transparent add-on to such protocols as 
10 TCP and UDP. 

Since, however, IPSEC decryption occurs at the IP layer, it can be 
difficult to port IPSEC to an application level gateway while still maintaining 
control at the proxy over authentication, message content, access control and 
auditing. Although the IPSEC specification in RFC 1825 suggests the use of a 
1 5 mandatory access control mechanism in a multi-level secure (MLS) network to 
compare a security level associated with the message with the security level of 
the receiving process, such an approach provides only limited utility in an 
appUcation level gateway environment. In feet, implementations on application 
level gateways to date have simply relied on the fact that the message was 
20 IPSEC-encrypted as assurance that the message is legitimate and have simply 
decoded and forwarded the message to its destination. Thb creates, however, a 
potential chink in the firewall by assuming that the encrypted communication 
has access to all services. 

What is needed is a method of handling PSEC messages within an 
25 application level gateway which overcomes the above deficiencies. The method 
should allow control over access by an IPSEC connection to individual services 
within the internal network. 

Summary th? TTT"'"''"*''?'! 
The present invention is a system and method for regulating the flow of 
30 messages through a firewall having a network protocol slack, wherein the 
network protocol stack includes an Internet Protocol (IP) l^er, the method 
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comprising the steps of determining, at the IP layer, if a message is encrypted, if 
tbc message is not encrypted, passing the unencrypted message up the network 
protocol stack to an ^Ucacion level proxy, and if the message is encrypted, 
decrypting the message and passing the decrypted message up the network 
protocol stack to the application level proxy, wherein the step of decrypting the 
message includes the step of executing a procedure at the IP layer to decrypt the 
message. 

According to another aspect of the present invention, a system and 
method is described for authenticating the sender of a message within a 
computer system having a network protocol stack, wherein the network protocol 
stack includes an Internet Protocol (IP) layer, the method comprising the steps of 
determining, at the IP layer, if the message is encrypted, if the message is 
encrypted, decrypting the message, wherein the step of decrypting the message 
includes the step of executing a procedure at the IP layer to decrypt the message, 
passing the decrypted message up the network protocol stack to an appHcation 
level proxy, determining an authentication protocol appropriate for the message, 
and executing the authentication protocol to authenticate the sender of the 
message. 

Brief Description of the T>rau;i^ g,ij 
20 In the following detailed description of example embodiments of the 

invention, reference is made to the accompanying drawings which form a part 
hereof, and wHch is shown by way of illustration only, specific embodiments in 
which the invention may be practiced. It is to be understood that other 
embodiments may be utilized and structaral changes may be made without 
25 departing from the scope of the present invention. 

In the drawings, where like numerals refer to Uke components throughout 
the sev^al views: 

Figure 1 is a fimctional block diagram of an application level gateway- 
implemented firewall-to-fircwaU encryption scheme according to the present 
30 invention; 



Figure 2 is a block diagram showing access control checking of both 
encrypted and unencrypted messages in network protocol stack according to the 
present invention; 

Figure 3 is a block diagram of a representative application level gateway- 
implemented firewall-to-firewall encryption scheme; 

Figure 4 is a block diagram of one embodiment of a network-separated 
protocol stack implementing IPSEC according to the present invention; and 

Figure 5 is a functional block diagram of a firewall-to-workstation 
encryption scheme according to the present invention. 

Description of the Preferred Prnhnriin,^ 
In the foUowing detailed description of the preferred embodiment, 
references made to the accompanying drawings which fonn a part hereof, and in 
which is shown by way of illustration specific prcfiarcd embodiments in which 
the invention may be practiced. These embodiments are described in sufficient 
detail to enable those skilled in the ait to practice the invention, and it is to be 
tmderstood that other embodiments may be utilized and that structural, logical, 
physical, architectural, and electrical changes may be made without departing 
firom the spirit and scope of the present invention. The following detailed 
description is, therefore, not to be taken m a limiting sense, and the scope of the 
present invention is defined only by the appended claims and their equivalents. 

A system 10 which can be used for fircwall-to-firewall encryption (FFE) 
is shown in Figure 1. In Figure I, system 10 includes a woikstation 12 
communicating through a firewaU 14 to an ux^rotectcd network 16 such as the 
Internet System 10 also includes a woikstation 20 communicating through a 
firewall 18 to unprotected network 16. In one embodiment, firewall 18 is an 
application level gateway. 

As noted above, IPSEC encryption and decryption work within the IP 
layer of the network protocol stack. This means that all communications 
between two IP addresses will be protected because all interfirewall 
communication must pass through the IP layer. IPSEC takes tiic standard 



Internet packet and converts it into a carrier packet. The carrier packet is 
designed to do two things: to conceal the contents of the original packet 
(encryption) and to provide a mechanism by which the receiving firewall can 
verify the source of the packet (authentication). In one embodiment of the 
present invention, each IPSEC carrier packet includes both an authentication 
header used to authenticate the sending machine and an enc^sulatcd payload 
containing encrypted data. The authentication header and the encapsulated 
payload feamres of IPSEC can. however, be used independently. As required in 
RFC 1825, DES-CBC is provided for use in encrypting the encapsulated payload 
while the authentication header uses keyed MD5. 

To use IPSEC, you must create a security association (S A) for each 
destination IP address. In one embodiment, each SA contains the Ibllowing 
information: 

Security Parameters Index (SPI) - The index used to find a SA on 
receipt of an IPSEC datagram. 

Destination IP address - The address used to find the S A and 

trigger use of IPSEC processing on output. 

The peer SPI - The SPI value to put on a IPSEC datagram on 

output. 

The peer IP address - The destination IP address to be put into the 

packet header if IPSEC Timnel mode is used. 

The Encryption Security Payload (ESP) algorithm to be used. 

The ESP key to used for decryption of input datagrams. 

The ESP key to used for encryption of output datagrams. 

The authentication (AH) algoriOm to be used. 

The AHkey to be used for validation of ii^ut packets. 

The AH k^ to be used for generation of the authcnlication data 

for ou^ut datagrams. 

The combination of a given Security Parameter Index and Destination IP 
address uniquely identifies a particular "Security Association/' In one 



embodiment, the sending firewall uses the sending userid and Deslination 
Address to select an appropriate Security Association (and hence SPI value). 
The receiving firewall uses flie combination of SPI value and Source address to 
obtain the appropriate Security Association. 

A security association is nonnaily one-way. An audienticated 
communications session between two fiiewalls will noraxally have two Security 
Parameter Indexes in use (one in each direction). The combination of a 
particular Security Parameter Index and a particular Destination Address 
uniquely identifies the Security AssociatioiL 

More information on the specifics of an EPSEC FFE implementation can 
be obtained firom the standards developed by the IPSEC work group and 
documented in Security Architecture for IP (RFC 1825) and in RFC's 1 826- 
1829. 

When a datagram is received from unprotected network 16 or is to be 
transmitted to a destination across unprotected network 16, flie firewall must be 
able to determine the algoriAms, keys, etc. that must be used to process the 
datagram correctly. In one embodiment, this information is obtained via a 
security association lookup. In one such embodiment, the lookup routine is 
passed several arguments: the source IP address if the datagram is being received 
from network 16 or the destination IP address if the datagram is to be transmitted 
across network 1 6, the SPI, and a flag thai is used to indicate whetlier the lookup 
is being done to receive or transmit a datagr am 

When an IPSEC datagram is received by firewall 18 firam unprotected 
network 16, the SPI and source IP address arc determined by looking in the 
datagram. In one embodiment a Security Association Database (SADB) stored 
withiiifirewalllSissearchcdfortheentrywidiamatchmgSPI. Inonesuch 
embodiment, security associations can be set up based on network address as 
well as a more granular host address. This allows the network administrator to 
create a security association between two firewalls with only a couple of lines in 
a configuration file on each machine. For such embodiments, the entry in the 
Security Association Database that has both the matching SPI and the longest 
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address match is selected as the SA cntiy. In another such embodiment, each S A 
has a prefix length value associated with the address. An address match on a SA 
entiy means fliat ibc addresses match for Ha number of bits specified by the 
prefix lengdi value. 

5 There are two exceptions to this search process. First, when an S A «itry 

is set marked as being dynamic it inqjlies that the user of this SA may not have a 
fixed IP address. In this case the match is fully deteimined by the SPI value. 
Thus it is necessary that the SPI values for such SA entries be unique in tlie 
SADB. The second exception is for SA entries marked as tunnel mode entries. 

10' In this case it is normally the case that the sending entity will hide its source 
address so that all that is visible on the public wire is the destination address. In 
this case, like in the case where the SA entries are for dynamic IP addresses, the 
search is done exclusively on the basis of the SPI. 

When transmitting a datagram across unprotected network 16 the SADB 

1 S is searched using only the destination address as an input hi Ms case the entiy 
which has the longest address match is selected and returned to the calling 
routine. 

In one OTibodiment, if firewall 1 8 receives datagrams which are 
identified as either an IP_PROTO_IPSEC^ESP or IP_PROTO_IPSEC_AH 

20 protocol datagram, diere must be a corresponding SA in the SADB or else 

firewall 1 8 will drop the packet and an audit message will be generated. Such an 
occurrence might indicate a possible attack or it might simply be a symptom of 
an erroneous key entry in the Security Association Database. 

In a system such as system 10, application level gateway firewall 1 8 acts 

25 as a buffer between unprotected network 1 6 and woricstations such as 

workstation 20. Messages commg ftom unprotected network 1 6 are reviewed 
and a determination is made as to whetiier execution of an authentication and 
identification protocol is warranted. In contrast to previous systems, system 1 0 
also performs this same determination on IPSEC-encrypted messages. If 

30 desired, the same authentication and identification can be made on messages to 
be transferred ftom workstation 20 to unprotected network 16. Figure 2 



aiustratcs one way of authenticating both enciypted and unencrypted messages 
in a system such as system 10. 

In the system of Figure 2 a network protocol stack 40 includes a physical 
layer 42, an Internet protocol (IP) layer 44, a Transport layer 46 and an 
application layer 48. Such a protocol stack exists, for instance on application 
level gateway firewall 18 of Figure 1. An application executing in application 
layer 48 can communicate to an application executing on another system by 
preparing a message and transmitting it tfarou^ one of the existing transport 
services executing on transport layer 46. Transport l^er 46 in turn uses a 
process executing in IP layer 44 to continue the transfer. Physical layer 42 
provides the software needed to transfer data through the communication 
hardware (e.g., a network interfece card or a modem). As noted above, IPSEC 
executes within IP layer 44. Encryption and authentication is transparent to the 
host as long as the network administrator has the Security Association Database 
correctly configured and a key management mechanism is in place on the 
firewall. 

In application level gateway firewall 18, aproxy 50 operating within 
application layer 48 processes messages transferred between internal and 
external networks. All network-to-network traffic must pass through one of the 
proxies within application layer 48 before being the transfer across networks is 
allowed. A message arriving from external network 16 is examined at IP layer 
44 and an SADB is queried to determine if the source address and SPI arc 
associated wifli an SA. In the embodiment shown in Figure 2, an SADB Master 
copy 52 is maintained in persistent memory at application layer 48 while a copy 
54 of SADB is maintained in volatile memory within the kernel. If the message 
is supposed to be encrypted, the message is decrypted based on the algorithm 
and key associated with the particular SA and the message is transferred up 
through transport layer 46 to proxy 50. Proxy 50 examines the source and 
destination addresses and the type of service desired and decides whether 
authentication of the sender is warranted. If so, proxy 50 initiates an 
authentication protocol. The protocol may be as simple as requesting a user 
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name and password or it may include a challenge/response authentication 
process. Proxy SO also looks to see whether the message coming in was 
encrypted or not and may &ctor that into whether a particular type of 
au&entication is needed. In Telnet for instance, user name/password 
5 authentication may he sufficient for an FFE link while the security policy may 
dictate that a more stringent challenge/response protocol is needed for 
unencrypted links. In that case, proxy SO will be a Telnet proxy and it will base 
its authentication protocol on whether the link was encrypted or not. 

Since IPSEC executes within IP layer 44 there is no need for host 

10 firewalls to update their applications. Users that already have IPSHC available 
on their own host machine will, however, have to request that the firewall 
administrator set up SA's in the SADB for their traflBc. 

In the embodiment shown in Figure 2, a working copy 54 of the Security 
Association Database consisting of all currently active SA's is kept resident in 

1 5 memory for ready access by IP layer processing as datagrams are received and 
transmitted. In addition, a working master copy S2 of the SADB is maintained 
in a file in nonvolatile memory. During system startup and initialization 
processing the content of all of the required SA's in master SADB 52 is added to 
Ae working copy 54 stored in kernel memory. 

20 In one embodiment, firewall 1 8 maintains different levels of security on 

internal and external network interfaces. It is desirable for a firewall to have 
different levels of security on bodi the internal and external interfaces. In one 
embodiment, firewall 18 supports tbree different levels^ numbered 0 through 2. 
These levels provide a simple policy mechanism that controls pemiission for 

25 both in-bound and out-bound packets. 

Level 0 - do not allow any in-bound or out-bound traffic unless there is a 
security association between the source and destination. 
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Level 1 - Allow both in-bound and out-bound non-IPSEC traffic but 
force the use of IPSEC if a SA exists for the address. (To support this firewall 
1 8 must look for a S A for each in^bound datagram.) 

Level 2 - allow NULL security associations to exist. NULL associations 
S are just like normal security associations, except no encryption or authentication 
transform is performed on in-bound or out-bound packets that correspond to this 
NULL association. With Level 2 enabled, the machine will still receive 
unprotected traffic, but it will not transmit unless Level 1 is enabled. 

The default protection level established when the Security Association 
10 Database (SADB) is initialized at boot time is 1 for in-bound trafiBc and 2 for 
out-bound traffic. 

An Access Control List, or ACL, is a list of rules that regulate the flow of 
Internet connections through a firewalL These rules control how a firewall's 
servers and proxies will react to coimection attempts. When a server or proxy 

1 S receives an incoming connection, it performs an ACL check on tfuit connection. 
An ACL check compares a set of parameters associated wiOi the 
connection against a list of ACL rules. The rules detennine whether the 
connection is allowed or denied. A rule can also have one or more side effects. 
A side effect causes the proxy to change its behavior in some fashion. For 

20 example, a common side effect is to redirect the destination IP address to an 
alternate machine. In addition to IP connection attempts, ACL checks can also 
made on the console logins and on logins made &om serial ix>rts. Finally, ACL 
checks can also be made on behalf of IP access devices^ such as a Cisco box, 
through the use of the industry standard TACACS+ protocol. 

25 In one embodiment, the ACL is managed by an acid daemon running in 

the kernel of firewalls 10 and 30. The acid daemon receives two types of 
requests, one to query the ACL and one to administer it. In one such 
embodiment^ the ACL is stored in a relational database such as the Oracle 
database for fast access. By using such a database, query execution is 

30 assmchronous and many queries can be executing concurrently. In addition, 
these types of databases are designed to manipulate long lists of rules quickly 
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and efficiently. These qualities ensure that a given query cannot hang up the 
process that issued the query for any appreciable time (> U2 seconds). 

In one such embodiment, the database can hold up to 100,000 iisers and 
up to 1 0,000 hosts but can be scaled up to the capacity of the underlying 
database engine. The results of an ACL check is cached, allowing repeated 
checks to be turned around very quickly. 

Applications on firewalls 10 and 30 can query acid to determine if a 
given connection attempt should be allowed to succeed. In one embodiment, the 
types of applications (i.e. "agents'*) that can make ACL queries can be divided 
into four classes: 

1) Proxies. These allow comiections to pass through firewall 10 or 30 in 
Older to provide access to a remote service. They include tnauthp 
(authenticated telnet proxy), pf^ (FTP proxy), httpp (HTTP proxy), and 
tcpgsp (TCP generic service proxy). 

2) Servers. These provide a service on the firewall itself. They include ftpd 
andhttpd. 

3) Login agents. Login agent is a program on the firewall that can create a 
Unix shell. It is not considered a server because it cannot receive IP 
connections. One example is /usr/bin/login ^en used to create a dialup 
session or a console session on firewall 10 or 30. Another example is the 
command srole, 

4) Network Access Servers (NAS). NAS is a remote IP access device, 
typically a dialup box manufactured by such companies as Cisco or 
Bridge. The NAS usually provides dialup telnet service and may also 
provide SLIP or PPP service. 

Proxies, servers, login agents, and NASes make queries to acid to 
determine if a given connection attempt should be allowed to succeed. All of the 
agents except NAS make their queries directly. NAS, because it is remote, must 
communicate via an auxiliary daemon that typically uses an industry standard 
protocol such as RADIUS or TACACS+. The auxiliary daemon (e.g., tacradd) 
in rum forwards the query to local acid. 
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As a side effect of tfie queiy, acid tells the agent if authentication is 
needed. If no authentication is needed, the connection proceeds immediately. 
Otherwise acid provides (as another side effect) a list of allowed authentication 
methods that the user can choose from. The agent can present a menu of choices 
S or simply pick the first authentication method by default. Typical authentication 
methods include plain password, SNK DSS, SDI SecurlD. LOCKout DBS, and 
LOCKout FORTEZZA. In one embodiment, the list of allowed authentication 
methods varies depending on the host name, user name, time of day, or any 
combination thereof. 

10 In the case of a Level 0 policy, it would be safe to assume that all 

incoming traffic is encrypted or authenticated. In the case of Levels 1 through 2, 
a determination must be made whether or not a security association exists for a 
given peer. Otherwise an ^plication n:iay believe that in-bound traffic has been 
authenticated when it really has not. (That is why it is necessary to look for an 

1 S S A on input of each non-IPSEC datagram.) 

In one embodiment^ a flag which accompanies the message as it is sent 
£rom IP layer 44 to proxy 50 indicates whether the incoming message was or was 
not encrypted. In another embodiment, proxy 50 accesses Security Association 
Database 54 (the table in the kernel can be queried via an SADB routing socket 

20 (PF-S ADB)) to detemune whether or not a security association exists for a given 
peer.. The SADB socket is much like a routing socket found in the stock BSD 
4.4 kernel (protocol family PF«-ROUTE) except that PF-SADB sockets are used 
to maintain the Security Association Database (SADB) instead of the routing 
table. Because &e private keys used for encryption, decryption, and keyed 

25 authentication are stored in this table, access must be strictly prohibited and 
allowed to only administrators and key management daemons. Cture must be 
taken when allowing \iser-level daemons access to /dev/mem or /dev/kraem 
as well, since the keys are stored in kernel memory and could be e3q>osed with 
some creative hacking. 

30 In one embodiment, a command-line tool called sadb is used to support 

the generation and maintenance of in-kemel version 54 of SADB, The primary 
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interface between this tool and the S ADB is the PF-SADB socket The kernel 
provides socket processing to receive client reqxiests to add, update, or change 
entries in in-kemel SADB 54. As noted above, the default protection level 
established when the Security Association Database (SADB) is initialized at boot 
5 time is 1 for in-bound traffic and 2 for out-bound traffic. This may be changed 
by the use of the sadb command. 

The existing sadb command was derived from the MIST implementation 
of IFSEC, As noted above, this tool is much like route in that it uses a special 
socket to pass data structures in and out of die kernel. Then are three commands 
10 recognized by the sadb command: get, set, delete. The following simple shell 
script supports adding and removing a single SA entry to SADB 54. Ii shows 
one embodiment of a parameter order for adding a SA to the SADB. 

# ! /bin/sh 
15 if [ $# -ne 1 ] 
then 

echo "usage: $0 <on>|<off>" >&2 
exit 1 

f i 

20 ONOFF-$l 
adds a ( ) 

{ 

IPADDRBSSs$2 
25 PEERADDRESS-0.0.0.0 

PREFIXLEN-0 # Num of bits, 0 => full 32 

bit: match 

LOCALADDRESS-0 .0.0.0 
REALADDRSSS-0 .0.0.0 
30 PORT=0 

PROTOCOL- 0 
UID-0 

DESALG=1 # I = DES-CBC 

IVLEN-4 # bytes 

35 DESKEY=ObObObObObObObOb 

DESKBYLENsS # bytes 

AHALG=1 # 1 = MD5 

AHKEY=3 0313233343536373031323334353637 

AHKEYLEN-ie # bytes 

40 LOCAL_5PI=$l 
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PEER_SPI=$1 

TUNNEL_MODE=0 

AHRESULTLEN=4 

COMBINED_MODE=l # On output, 1 = ESP, then 

5 AH; 0 -> AH, then ESP 
DyNAMIC_FLAG»0 

if [ "$ONOFF" « "on" 
then 

10 ./sadb add dat $IPADDRESS SPREFIXLEN $LOCAL_SPI 

$UID $PBERADDRESS $PEBR_SPI $TUNNEL_MODE $LOCAlADDRESS 
$REAIADDRBSS $PROTOCOL $PORT $DESALG $IVLEN $DESKEYLEN 
SDESKEY $DESKEYLEN $DESKEY $AHAL6 $AHKEYIiEN $AHKEY 
$AHKEyLEN $AHKEy $AHRESULTIiEN ?CX5MBINED_M0DE 

15 $DyNAMIC_FIAG 
else 

./sadb delete dst $IPADDRESS $LOCAL-SPI 

fl 
} 

20 

# Get down to work: 

addsa 500 172.17.128-115 # nuniber6.sctc.com 

The cuiTCTt status of in-lKmel SADB 54 can be obtained with Has sadb 
25 conunand. The get option aUows dumping the entire SADB a single entry. In 
one embodiment, the complete dump approach uses /dev/kmem to find the 
infonnation. The infoimation may be presented as follows: 



# sadb get dst 

30 

Local -SPI Address-Patnily Destination-Addr 
Pref lx_length UID 

Peer -Address Peer-SPI Transport -Type 
Local -Address Real -Address 
35 Protocol Port 

ESP_Alg_ID ESP_IVEC_Length 

ESP_Enc_Key_length ESP_Enc_ESP__Key 
ESP__Dec_Key_length ESP__Dec_ESP~Key 
AH_Alg_ID AH_Data_Length 
40 AH_Gen_Key_Length AH_Gen_Key 

AH_Check_Key_Length AH_Check_Key 
Combined^Mode DyTiamic_Flag 
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500 INET: nuniberS . sctc . com 0 0 
0.0.0.0 500 Transport (0) 0 
0,0.0.0 0.0.0.0 
None None 

DES/CBC-RFC1829 (1) 4 

8 ObObObObObObObOb 
8 ObObObObObObObOb 
MD5-RPC1828.(1) 4 

16 30313233343536373031323334353637 
16 30313233343536373031323334353637 
ESP+AH(1) 0 

501 INET: spokes . sctc . com 0 0 
0.0.0.0 501 Transport (0) 0 
0,0.0.0.0.0.0-0 
None None 

DES/CBC-RFC1S29 (1) 4 

8 ObObObObObObObOb 
8 ObObObObObObObOb 
MD5-RFC1828(1)4 

16 30313233343536373031323334353637 
16 30313233343536373031323334353637 
ESP-)-AH(l) 0 

End of list. 

When a new entiy is added to in-konel S ADB 54, the add process first 
checks to see that no existing entry will match the values provided in the new 
30 entry. If no match is found then the entry is added to the end of the existing 
SaDB list. 

To illustrate the use and administration of an FFE, we'll go through an 
example using FFE 70 in Figure 3, Firewalls 14 and 18 are ha&. application 
level gateway fizewalls in^Iemented according to liie present invention. 
3S Workstations H2 and H3 both want to communicate with HI. For the 
»lininistrator of firewalls 14 and 18, this is easy to accomplish. The 
administrator sets up a line something like this (we'll only show die IP address 
part and SPI parts of the SA, since they're the trickiest values to configure. Also, 
assume that we are using tunnel mode): 

40 

# Hypothetical SWl Config File 
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15 



20 



25 
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# 

# Fields are laid out in the following manner: 

# srcaddrornet" localSPI= peeraddr« peerSPIs= 
realsrcaddr= localaddr= key« 

5 

# The following entry sets up a tunnel between hosts 
behind SWl 

# and hosts behind SW2, 

src=172.16.0,0 localSPI-666 peer=192 • 168 • 100 . 5 
10 peerSPl«777 \ 

realsrcaddr=192 . 168 . 100 . 5 localaddrs=0 ,0.0.0 
keysOxdeadbeef fadebabe 

# Hypothetical SW2 Config Pile 
15 # 

# Fields are laid out in the following manner: 

# srcaddromet= localSPI= peeraddr^ peerSPI^ 

realsrcaddr= localaddr* key= 

20 # The following entry sets up a tunnel between hosts 
behind SWl and 

# hosts behind SW2 . 

src=172*17.0.0 localSPI«777 peer=192 . 168 , 20 . 1 
peerSPI-666 \ 
25 realsrcaddr=192 .168.20.1 localaddr=0 , 0 . 0 . 0 \ 

keysOxdeadbeeff adebabe 

With diis setup, ail traffic is encrypted using one key, no matter 'wbo is 
talking to ^om. For example^ trafBc fiom H2 to HI as well as traffic fix>m H3 

30 to HI will be encrypted with one key. Although this setup is small and simple, it 
may not be enough. 

What happens if H2 cannot trust H3? In tins case, the administrator can 
set up security associations at the host level. In this case, we have to rely on the 
SPI field of the SA, since the receiving firewall cannot tell from tlie datagram 

35 header which host behind the sending firewall sent the packet. Since the SPI is 
stored in IPSEC datagrams, we can do a lookup to obtain its value. Below are 
the sample configuration files for both firewalls again, but this time, each host 
combination communicates with a different key. Moreover, H2 excludes H3 
from communications with Hl» and H3 excludes H2 in the same way. 

40 
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# Hypothetical SWl Config File 
# 

# Fields are laid out in the following manner: 

# srcaddromct= localSPI= peeraddr- peerSPI* 
5 realsrcaddr= localaddr= key= 

# The following entry sets up a secure link between H2 
and HI 

src-172 . 16 . 0 . 2 localSPI-666 peer-192 . 166 . 100 . 5 
10 peerSPI=777 \ 

realsrcaddr-192 . 168 . 100 . 5 
localaddrs=l78.17.128.7i \ 

keyaOxOaOaOaOaOaOaOaOa 

15 # The following entry sets up a secure link between H3 
and HI 

src-l72.16.0.1 localSPI=555 peer-192 . 168 . 100 . 5 
peerSPI=888 \ 

realsrcaddrsl92 .168.100.5 
20 localaddrs=178.l7,128.71 \ 
keysOxObObObObObObObOb 

# Hypothetical SW2 Config File 

25 # Fields are laid out in the following manner: 

# srcaddrornet= localSPI= peeraddr- peerSFI* 
real8rcaddr<° localaddr" key- 

# The following entry sets up a secure link between H2 
30 and HI 

src=l72 .17.128.71 localSPI=777 peer-192 . 168 . 20 - 1 
peerSFls666 \ 

realsrcaddr-192. 168 .20.1 localaddrsBi72 . 16 . 0 . 2 \ 

keysOxOaOaOaOaOaOaOaOa 

35 

# The following entry sets up a seciire link between H3 
and HI 

src=172.17.128.71 localSPI-888 peer=192 , 168 . 20 . 1 
peerSPI-555 \ 

40 realsrcaddr»192.168.20.l localaddrB«l72 . 16 , 0 . 1 \ 

key- OxObObObObObObObOb 

Figute 4 is a block diagram showing in more detail one embodiment of 
an IFSEC-enabled application level gateway firewall 1 8. Application level 
45 gateway firewall 1 8 provides access control checking of both encrypted and 
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unencrypted messages in a more secure enviromnent due to its network- 
separated architecture. Network separation divides a system into a set of 
indqjendent regions or burbs, with a domain and a protocol stack assigned to 
each burb. Each protocol stack 40x has its own independent set of data 

5 structures, including routing infomiation and protocol information. A given 
socket will be bound to a single protocol stack at creation time and no data can 
pass between protocol stacks 40 without going through proxy space. A proxy 50 
therefore acts as the go-between for transfers between domains. Because of this, 
a malicious attacker who gains control of one of the regions is prevented from 

10 being able to compromise processes executing in other regions. Network 
separation and its application to an application level gateway is described in 
"SYSTEM AND METHOD FOR ACHIEVING NETWORK SEPARATION'', 
U.S. Application No. 08/599,232, filed February 9, 1996 by Gooderum ct al. 
In the system shown in Figure 4, the in-bound and out-bound datagram 

1 5 processing of a security association continues to follow the conventions defined 
by the network separation model. Thus aU datagrams received on or sent to a 
given burb remain in that burb once decrypted. In one such embodiment S ADB 
socket 78 has been defined to have the type *sadb\ Each proxy 50 that requires 
access to SADB socket 78 to execute its query as to whether the received 

20 message was enciypted must have create permission to the sadb type. 

The following is list of specific requirements that a system such as is 
shown in Figure 4 must provide. Many of the requirements were discussed in 
the information provided earlier in this document 

1 . Firewall applications may query the IPSEC subsystem to determine if 
25 traffic with a given address is guaranteed to be encrypted. 

2. Receipt of an unencrypted datagram from an address that has a S A results 
in the datagram being dropped and an audit message being generated. 

3. On receipt of encrypted protocol datagrams the SADB searches will be 
done using the SPI as the primary key. The source address will a 

30 secondary key. The SA returned by the search will be the SA which 

matches the SPI exactly and has the longest match with the address. 
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4. A search of the SADB for a SPI that finds an entry that is marked as SA 
for a dynamic IP will not consider the address in the search process. 

5. A search of the SADB for a SPI that finds an entry that is marked as a SA 
for a tunnel mode connection -will to consider the address if it is (0.0.0.0) 
i.elNADDR. 

6. On receipt of a non-IPSEC datagram the SADB wUl be searched for an 
entry that matches the src address. If a SA is found the datiigram -will be 

dropped and an audit message sent 

7. SADB searches on ou^ut will be done using the DST address as key. If 

10 more than one SA entry in the SADB has that address the fust one with 

the maximum address match will be returned. 

8. The SADB must be structured so that searches are fest regardless if the 
search is done by SPI or by address. 

9. The SADB must provide support for connecfions to a site with a fixed 
15 SPI but changing IP address. SA entries for such connections will be 

referred to as Dynamic Address Sites, or just Dynamic emries. 

10. When a dynamic entry is found by a SPI search, the current datagram's 
SRC address, which is required to ensure that Ac return datagrams are 
properly encrypted, will be recorded in the SA only after the AH 

20 checking has passed successfully. (This is because if the address is 

recorded before AH passes thwi an attacker can cause return packets of 
an outgoing connection to be transmitted in the dear.) 

11. A failure of an AH check on a dynamic entry results in an audit message. 

12. In an cmbodimHit where the firewall requires that all comlections use 
25 both AH and ESP, on receipt Reorder should be AH first ESP second. 

1 3 . The processing structure on both input and output should try to minimize 
Hit number of SADB required lookups. 



30 



Returning to Figure 4, in one embodiment firewall 18 includes a crypto 
engine interfece 80 used to encrypt an IPSEC payload. Crypto engine interface 
80 may be connected to a sofiware encryption engine 82 or to a hardware 
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encryption engine 84. Engines %2 and 84 perfbrai the actual encryption function 
using, for example, DES-CBC In addition, software encryption engine 82 may 
include the keyed MD5 algorithm used for AR 

In one embodiment, crypto engine interface 80 is a utUity which provides 
5 a consistent interfece between the software and hardware encryption engines. As 
shown in Figure 4, in one such embodiment intcrfece 80 only supports the use of 
the use of hardware cryptographic engine 84 for IPSEC ESP processing. The 
significant design issue that interface 80 must deal with is that use of a hardware 
encryption engine requires that the processing be down m disjoint steps 
1 0 operating in different interrupt contexts as engine 84 completes the various 
processing steps. 

The required information is stored in a request structure that is bound to 
the IP datagram bang processed. The request is of type crypto^r equest_t. 
This structure is quite large and d^initely does not contain a minimum state set. 
1 5 In addition to the definition of the request data stnicnire, this software 

implementing interface 80 provides two fimctions which isolate tlie decision of 
which cryptographic engine to use. The crypt_des_encrypt fimction is for 
use by the IP outpxst processing to encrypt a datagram. The 
crypt_des_decryptL function is for use by the IP input processing to 
20 decrypt a datagram. If hardware oicryption engine 84 is present and other 

hardware visage criteria are met the request is oiqueued on a hardware processing 
queue and a return code indicating that the cryptographic processing is in 
progress is returned. If software engine 82 is used, the return code indicates thai 
the cryptographic processing is complete. In the former case, the continuation of 
25 the IP processing is delayed until after hardware encryption is done. Otherwise 
it is completed as immediately in die same processing stream. 

There are two software cryptographic engines 82 provided in the IPSEC 
software. One provides the MD5 algorithm used by the IPSEC AH processing, 
and the other provides the DES algorithm used by the IPSEC ESP processing. 
30 This software can be obtained from the US Government IPSEC implementation. 
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In one embodiment hardware ciyptographic engine 84 is provided by a 
Cylink SafeNode processing board. The interface to Hus hardware card is 
provided by the Cylink device driver. A significant aspect of the Cylink card 
that plays a major part in the design of the IPSEC Cylink driver is ihat the card 

5 functions much like a low level subroutine interfece and requires software 

support to initiate each processing step. Thus to encrypt or decrypt an individual 
datagram there are a minimum of two steps, one to set the DES initialization 
vector and one to do the encryption. Since the IP processing can not suspend 
itself and wait while the hardware completes and Aen be rescheduled by the 

1 0 hardware interrupt handler, in one embodiment a finite state machine is used to 
tie sequences of hardware processing elements together. In one such 
embodiment the interrupt handler looks at the current state, executes a defined 
after state function, transitions to the state and then executes that state' s start 
function. 

1 5 One fimction, cyl_enqueue_reques t , is used to initiate cither an 

encrypt or a decrypt action. This fimction is designed to be called by 
cryptographic engine interface 80. All of the information required to initiate the 
processing as well as the fimction to be performed after the encryption operation 
is completed is provided in the request strocmre. This function will enqueue the 

20 request on the hardware request queue and start the hardware processing if 
necessary. 

A system 30 which can be used for firewall-to-workstalion encryption is 
shown in Figure 5. In Figure 5, system 30 includes a workstation 12 
communicating through a firewall 14 to an unprotected network 16 sudi as the 

25 Internet. System 30 also includes a workstation 32 coromunicaiing directly with 
firewall 14 through unprotected network 16. Firewall 14 is an application level 
gateway incoiporaring IPSEC handling as described above. Qx should be noted 
that IPSEC security cannot be used to authenticate die personal identity of the 
sender for a firewaU to firewall transfisr. When ff SEC is used, however, on a 

30 single user machine such as a portable personal computer, IPSEC usage should 
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be protected with a personal identification mmiber (PIN). In ^ cases IPSEC 
can be used to help with user identification to the firewall.) 

According to the IPSEC RFC's, you can use cither tunnel or transport 
mode with this embodiment based on your security needs. In certain situations, 
the conimunications must be sent in tunnel mode to hide unregistered addresses. 

Although specific embodiments haw been illustrated and described 
herein, it will be appreciated by those of ordinary skiU in the art that any 
arrangement which is calculated to achieve the same purpose may be substituted 
for the specific embodiment shown. This application is intended to cover any 
adaptations or variations of the present invention. Therefore, it is intended that 
this invention be limited only by the claims and the equivalents thereof. 
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What is claimed is: 

1 AmedujdofreguladngtheflowofmessagesthroughafirewallhaviBga 
net«.ork protocol stack, wherein ^e network protocol stack includes an Internet 
5 Protocol (IP) layer, the method compiiang the steps of: 

detcnnining. at the IP layer, if a message is encrypted; 
if the message is not encrypted, passing the unencrypted message up the 
network protocol stack to an appUcation level proxy; and 

if the message is encrypted, decrypting die message and passing the 
1 0 decrypted message up the network protocol stack to th'e application level proxy, 
wherein the step of decrypting the message includes the step of executing a 
procedure at Ae IP layer to decrypt the message. 

2. A method of authenticating the sender of a message within a computer 
15 systemhavinganetworkpTOtocolsta«*.whereinthenetworkprotocolstack 

includes an Internet Protocol (IP) layer, the method comprising the steps of: 
determining, at thcIP layer, if the message is encrypted; 
if the message is encrypted, decrypting the message, wherein the step of 

decrypting the message includes the step of executing a process at the IP layer to 

20 decrypt the message; 

passing the decrypted message up the network protocol -stack to an 

application level proxy; 

determining an authentication protocol appropriate for the message; and 
executing the authentication protocol to authenticate the sender of the 
25 message. 

3. The method according to claim 2 wherein the step of determining an 
authentication protocol appropriate for the message includes the steps of: 
determining a source IP address associated with the message; and 
30 determining the authentication protocol associated with the source IP 

address. 
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4. The method according to claim 2 wherein the message includes secmity 
j«^eters index and v^erein the step of determining an authentication protocol 
appropriate for lie message includes the steps of: 

detcnnining the authentication protocol associated with a dynamic IP 
address, wherein the step of determining the anihcnticationptotDcol includes the 
step of looking up a security association based on the security parameters index; 

determining a current address associated with the dynamic source IP 
address; and 

binding the current address to the security parameters index. 

5, A firewall, comprising: 

a first commxmications interface; 
a second communications imcrfece; 

a network protocol stack connected to the first and the second 
15 communications interfaces, whereinthenetworkprotocolstackincludes an 

Internet Protocol Q.?) layer and a transport layer, 

a decryption procedure, operating at the IP layer, wherein the decryption 
procedure decrypts encrypted messages received at one of said first and second 
communications interfeces and ouq)uts decrypted messages; and 
20 aproxy,connectedtothetransportUycrof saidoetworkprotocol stack, 

wherein the proxy receives decrypted messages ftom the decryption procedure 
and executes an authentication protocol based on the content of the decrypted 

message. 

25 6. A firewall, comprising: 

a first communications interface; 
a second communications interface; 

a first network protocol stack connected to the first communications 
interface, v*ereir» the first network protocol stack includes an Internet Protocol 
30 GP) layer and a transport layer; 
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a second network protocol stack connected to the second 
communications interface, wherein the second network protocol steick includes 
an Internet Protocol QP) layer and a transport layer; 

a decryption procedure, operating at the IP layer of the first network 
5 protocol stack, the decryption procedure lecciving encrypted messages received 
by said first communications interface and ouiputdng decrypted messages; and 

a proxy, connected to the transport layers of said first and second network 
protocol stacks, the proxy receiving decrypted messages from the decryption 
procedure and executing an authentication protocol based on the content of the 
10 decrypted message. 

7. The firewall according to claim 6 wherein the firewall further includes: 
a third communications inter&ce; and 

a third network protocol stack connected to the third communications 
15 interface and to the proxy, wherein the third network protocol stack includes an 
Internet Protocol (IP) layer and a transport Uycr and wherein the second and 
third network protocol stacks are restricted to first and second butbs, 
respectively. 

20 8. A method of estabUshing a virtual private network between a first and a 
second network, ^**ercin each network includes an appUcation level gateway 
firewan which uses a proxy operating at the appUcation layer to process traffic 
through the firewall, wherein each firewall includes a network protocol stack and 
wherein each network protocol stack includes an Internet Protocol (IP) layer, the 

25 method comprising the stqps of: 

transferring a connection request from the first network to the second 

network; 

determining, at thcIP layer of the network protocol stack of the second 
network's firewall, if the connection request is encrypted; 
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if the connection request is encrypted, decrypting the request, wherein the 
step of decrypting the request includes the step of executing a procedure at the IP 
layer of the second network's firewall to decrypt the message; 

passing the connection request up the network protocol stack to an 

5 application level proxy; 

determining an authenticalion protocol appropriate for the connection 

request; 

executing the authentication protocol to authenticate the connection 
request; and 

10 if the connection request is authentic, establishing an active connection 

between the first and second networks. 

9. The method according to claim 8 wherein the step of executing the 
authentication protocol includes the step of executing program code within the 

1 5 firewall of the second network to mimic a challenge/response pn)tocol executing 
on a server internal to the second network. 

10. The method according to claim 8 wherein the step of executing the 
authentication protocol includes the step of executing program code to execute 

20 the authentication protocol in line to the session. 

11. The method according to claim 8 wherein the step of detMmining an 
authentication protocol includes the step of determining if the connection request 
arrived encrypted and selecting the authentication protocol based on whether tiie 

25 connection request was encrypted or not encrypted. 
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